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ABSTRACT:  In  the  current  warfighting  environment,  the  military  needs  robust  modeling  and  simulation  (M&S)  to 
support  Irregular  Warfare  (IW)  analysis  across  the  range  of  tactical,  operational,  and  strategic  levels  of  warfare  to 
help  inform  decisions  concerning  operations  within  the  IW  environment.  In  support  of  this  need,  the  military  requires  a 
responsive  family  of  Models,  Methods,  and  Tools  (MMT)  able  to  credibly  represent  US  and  Coalition  ground  forces 
conducting  operations  in  a  Joint  and  Combined  IW  environment,  from  the  tactical  to  strategic  levels.  As  a  first  step  in 
this  direction,  TRAC  Monterey  (TRAC-MTRY)  is  developing  a  prototype  capability  that  credibly  represents  ground 
forces  conducting  IW  operations  and  focusing  on  the  relevant  relationships  and  interactions  within  the  population. 
This  paper  describes  work  being  performed  on  behalf  of  TRAC-MTRY  to  develop  a  measurable,  repeatable  method  for 
assessing,  understanding,  and  describing  the  risk  of  using  an  M&S  for  analysis,  to  enhance  the  ability  of  decision 
makers  to  assess  the  risk  in  using  an  IW  M&S,  and  add  to  the  core  body  of  knowledge  in  Validation  Best  Practices. 


1.  Introduction 

In  the  current  warfighting  environment,  the  military  needs 
robust  modeling  and  simulation  (M&S)  to  support 
Irregular  Warfare  (IW)  analysis  across  the  range  of 
tactical,  operational,  and  strategic  levels  of  warfare  to 
help  inform  decisions  concerning  operations  within  the 
IW  environment.  Violent  extremist  networks,  which  are 
tactful,  complex  adaptive  systems  with  the  outward 
appearing  ability  to  act  without  direction  are  implicit 
within  IW.  Appropriate  and  meaningful  responses  to 
these  violent  extremist  networks  require  understanding  of 
the  underlying  population,  its  dynamics,  and  its  driving 
forces.  In  support  of  this  need,  the  military  requires  a 
responsive  family  of  Models,  Methods,  and  Tools  (MMT) 
able  to  credibly  represent  US  and  Coalition  ground  forces 
conducting  operations  in  a  Joint  and  Combined  IW 
environment,  at  the  tactical  to  strategic  levels.  As  a  first 
step  in  this  direction,  TRAC  Monterey  (TRAC-MTRY)  is 
developing  a  prototype  capability  that  credibly  represents 
ground  forces  conducting  IW  operations  and  focusing  on 


the  relevant  relationships  and  interactions  within  the 
population.  To  this  end,  TRAC-MTRY  has  developed  the 
Cultural  Geography  Model  (CGM),  a  government  owned, 
open  source  multi-agent  system  utilizing  Bayesian 
networks,  queuing  systems,  the  Theory  of  Planned 
Behavior,  and  Fischer’s  Narrative  Paradigm,  as  a  first  step 
in  the  development  of  a  family  of  models  to  support  the 
defense  analyst  in  answering  questions  relevant  to  IW 
such  as  “Is  security  adequate?”,  “Will  the  outcome  of 
upcoming  elections  be  legitimate?”  or  “Will  the  presence 
of  troops  increase  civilian  violence?”  with  responses 
similar  to  polling  data  (Alt  et  al  2009  -  JDMS  pre-pub 
copy).  Effective  validation  of  models  within  this  context 
requires  progress  in  the  theory  of  validation.  This  paper 
reports  on  the  necessary  background  required  to  support 
work  being  performed  on  behalf  of  TRAC-MTRY  to 
develop  a  measurable,  repeatable  method  for  assessing, 
understanding,  and  describing  the  risk  of  using  an  M&S 
for  analysis,  to  enhance  the  ability  of  decision  makers  to 
assess  the  risk  in  using  an  IW  M&S,  and  add  to  the  core 
body  of  knowledge  in  Validation  Best  Practices. 
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2.  Modeling  IW 

The  M&S  of  IW  requires  the  development  of  new  M&S 
methods.  The  social  science  on  which  this  development 
hinges  is  in  its  infancy.  In  particular,  the  social  science  is 
often  biased  by  western  perspectives  in  many  areas; 
includes  multiple  theories  to  describe  the  same 
phenomena,  often  uncorrelated  and  sometimes 

contradictory;  and  lacks  empirical  data  and  underlying 
computable,  mathematical  structures  to  inform  and 

validate  modeling  efforts.  In  fact,  the  data  that  is 

available  is  often  qualitative  vice  quantitative  and  the 
relationships  between  available  quantitative  data  and  its 
effects  on  the  social  systems  of  interest  are  unknown  (e.g., 
the  human  engagement  that  occurs  between  military  units 
and  the  population,  and  its  mutual  relationship  with 
DIME/PMESII  at  higher  levels  over  time).  Even  in  well 
understood,  homogeneous  populations,  population 

modeling  is  difficult  because  of  the  complexity  of  human 
cognition.  Heterogeneous,  unfamiliar  populations  only 
exacerbate  this  problem.  A  method  is  needed  to  assess 
the  available  data,  social  science,  and  the  developed  M&S 
in  a  measurable,  repeatable  way  for  assessing, 
understanding,  and  describing  the  risk  of  using  an  M&S 
for  analysis.  Development  of  this  risk  assessment  method 
is  a  key  element  in  Validation  Best  Practices. 

2.1  Validating  IW  models 

The  DoD  guidance  for  accomplishing  VV&A  is  well 
known  and  documented.  While  results  validation  and 
face  validation  are  often  used  methods  for  the  validation 
of  models,  the  difficulties  with  this  approach  for 
simulations  having  sensitivity  to  initial  conditions, 
chaotic,  or  emergent  effects,  and  the  difficulties  with 
validating  human  based  representation  models  is  well 
known  (Harmon  et  al.  2002,  Defense  Modeling  and 
Simulation  Office  2006,  Akst  2006,  Moya  et  al.  2007). 
The  validation  literature  consists  mainly  of  validation 
approaches,  paradigms,  and  techniques  as  well  as  specific 
validation  applications  and  assessments.  There  is  no 
mechanism  guiding  the  appropriate  selection  of  approach 
and  techniques  in  a  given  M&S  application.  Progress  is 
required  that  will  lead  to  effective  validation,  supporting 
the  need  for  developing  “fundamental  new  approaches  of 
conducting  VV&A  ...  [and]  ...  developing  new  VV&A 
methods  and  techniques  . . .  [with]  practical  value” 
(Sargent  et  al.  2000). 

To  address  this  need,  the  Marine  Corps  Combat 
Development  Center  (MCCDC)  Operations  Analysis 
Division  (OAD)  commissioned  an  Agent  Based 
Simulation  (ABS)  Verification,  Validation,  & 
Accreditation  (VV&A)  Framework  Study  in  2008  to 


develop  general,  institutionally  acceptable  processes  and 
criteria  for  assessing  the  validity  of  agent-based 
simulations  used  as  part  of  DoD  analyses  with  a  focus  to 
IW  analyses.  At  its  onset,  this  study  focused  on  the 
concept  of  validity,  viewing  the  verification  process  for 
simulation  as  the  same  as  for  software  verification  and 
accreditation  as  an  agreement  between  analysts  and  the 
study  sponsor  that  a  particular  model  is  useful  for  a 
particular  analysis  problem.  It  addressed  the  verification 
and  accreditation  processes  with  respect  to  their 
interdependencies  with  the  validation  process. 

The  MCCDC  OAD  effort  focused  on  the  validation  of  the 
non-physics  based  aspects  of  the  validation  problem  with 
the  goal  to  maintain  the  analytic  rigor  of  the  traditional 
VV&A  process,  while  expanding  it  to  cover  non- 
traditional  topics  (e.g.,  population  dynamics  and  cultural 
shifts).  The  effort  demonstrated  the  validation  process  of 
ABS  in  two  applications  to  guide  the  development  of  a 
framework  that  would  provide  a  means  for  assessing  the 
reliability,  applicability  and  feasibility  of  the  ABS  for  its 
intended  use,  preferably  in  a  quantifiable  way  for  future 
validation  efforts.  A  key  finding  of  this  work  is  that  the 
validation  of  an  M&S  for  analysis  cannot  be  decoupled 
from  that  analysis.  The  effort  for  TRAC-MTRY  will 
leverage  and  expand  on  the  MCCDC  OAD  effort  in  an 
applied  way. 

2.2  CGM  validation  project 

The  DoD  requires  robust  IW  modeling  in  the  current 
environment.  TRAC-MTRY  is  developing  capabilities  to 
help  determine  the  potential  impact  of  culture  and  the 
actions  of  the  civilian  population  on  current  operations. 
As  part  of  this  larger  effort,  it  is  essential  to  have  a 
validated  conceptual  model  underlying  the  CGM 
reflective  of  the  selected  social  science  underpinnings. 
This  project  will  develop  a  measurable,  repeatable  method 
for  assessing,  understanding,  and  describing  the  risk  of 
using  an  M&S  for  IW  analysis  as  well  as  develop 
validation  methodologies  for  assessing  the  CGM 
conceptual  model  and  implementation  (Figure  2.1).  It  has 
the  objective  to  assess  the  operational  utility  of  the  CGM 
with  suggestions  for  its  analytical  use  that  make  the 
operational  utility  accessible  and  mitigate  any  issues 
within  the  uses  of  interest.  It  supports  Key  Tenets  of  the 
TRAC  IW  Campaign  plan  by  enabling  an  incremental 
development  cycle,  with  interim  proof-of-principle  and 
prototype  applications  (“build-use-leam-fix”  approach) 
and  fits  within  the  MMT  line  of  effort  by  supporting  the 
development  of  a  Validation  and  Verification  (V&V) 
methodology  that  helps  achieve  useable  capabilities  as 
fast  as  acceptable  risk  and  resourcing  permit. 
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Figure  2.1.  Problem  Context 


3.  Validating  Human  Behavior  Models 

The  validation  of  IW  M&S  for  analysis  lies  within  the 
intersection  between  the  spheres  of  VV&A,  IW,  and  Risk 
as  shown  in  Figure  2.1.  Developing  core  knowledge  of 
the  IW  is  the  purview  of  our  military  specialists.  The 
question  of  how  VV&A  may  be  applied  within  the  IW 
sphere  has  been  asked  (reference  to  be  added).  Questions 
arising  from  the  intersection  of  the  VV&A  and  risk 
spheres  are  more  often  well-understood  for  physics-based 
or  engineering  models  but  less  frequently  so  for  M&S 
techniques  such  as  agent-based  simulation.  The 
intersection  of  the  risk  and  IW  spheres  is  the  domain  of 
the  art  of  warfare  and  out  of  scope  for  the  technical 
discussion.  The  addition  of  risk  to  the  analysis  allows  a 
more  formal  discussion  of  the  usefulness  and  limitations 
of  M&S  derived  information.  Our  focus  is  on  the 
innermost  intersection  where  these  questions  may  be 
answered  in  a  real  way  for  the  IW  problem. 

3.1  Validation  importance 

Acceptability  and  usability  get  at  the  key  points  for  why 
validation  is  important:  to  establish  the  credibility  of  a 
simulation  for  a  specified  intended  use  (Modeling  and 
Simulation  Coordination  Office  2004b).  This  includes 
determining  that  the  simulation  is  correct  and  meets 
requirements  through  software  engineering  and  other 
processes  but  is  not  limited  to  that.  It  also  includes 
providing  users  with  sufficient  information  to  determine  if 
the  simulation  can  meet  their  needs  as  well  as  determining 
the  simulation’s  capabilities,  limitations,  and  performance 
relative  to  the  real-world  objects  it  simulates.  User 


participation  throughout  the  development  process 
facilitates  this  confidence. 

The  DoD  guidance  for  accomplishing  VV&A  is  well 
known  and  documented.  While  results  validation  and 
face  validation  are  often  used  methods  for  the  validation 
of  models,  the  difficulties  with  this  approach  for 
simulations  having  sensitivity  to  initial  conditions, 
chaotic,  or  emergent  effects,  and  the  difficulties  with 
validating  human  based  representation  (HBR)  models  is 
well  known  (Harmon  et  al  2002,  Modeling  and 
Simulation  Coordination  Office  2004b,  Akst  2006,  Moya 
et  al  2008). 

Understanding  the  validity  of  the  M&S  of  physics  based 
and  engineering  systems  for  a  given  use  is  well 
understood.  Further,  physics-based  combat  models  have 
a  long  history  of  use.  However,  the  M&S  of  IW  requires 
the  development  of  new  M&S  methods.  Further,  the 
social  science  on  which  this  development  hinges  is  in  its 
infancy.  In  particular,  the  social  science  is  often  biased 
by  western  perspectives  in  many  areas;  includes  multiple 
theories  to  describe  the  same  phenomena,  often 
uncorrelated  and  sometimes  contradictory;  and  lacks 
empirical  data  and  underlying  computable,  mathematical 
structures  to  inform  and  validate  modeling  efforts. 

3.2  Necessary  elements  for  HBR  validation 

The  robust  documentation  of  the  conceptual  model; 
testing;  and  the  theoretical  support,  traceability  and 
justification  for  assumptions  facilitate  user  confidence. 
Using  a  well-defined,  documented  validation  process 
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supports  model  credibility.  Using  strong  validation 
methodologies  ensure  that  models  are  built  on  a  solid 
framework  of  standardized  organization,  process, 
products,  and  techniques;  and  that  they  simulate  accurate, 
consistent,  and  reproducible  results.  Without  strong, 
documented  methodologies,  valid  simulations  may  be 
rejected,  invalid  outcomes  may  be  accepted,  or 
simulations  may  be  used  improperly  (e.g.,  outside  of 
intended  use  or  in  opposition  to  embedded  assumptions). 
Formal  methods  allow  for  the  precise  description  of  a 
simulation’s  capabilities.  Further,  the  ability  to  make 
general  statements  about  individual,  general,  and 
federated  models  facilitates  use  and  re-use  of  those 
models. 

Any  effective  validation  methodology  needs  to  have  the 
following  characteristics  (Weisel  and  Moya  2007): 

1)  Transparent  -  to  provide  an  understanding  of  the 
assumptions,  decisions,  and  activities  that  went  into 
V&V  (I  know  what  I  have) 

2)  Traceable  -  to  ensure  the  flow  of  activities  and 
actions  is  logical  and  that  appropriate  referents  for 
those  activities  can  be  located  and  consulted  (I  know 
where  I  got  it) 

3)  Reproducible  -  to  provide  for  the  event  that  the  same 
model/data/users  will  be  applied  to  a  similar  effort  in 
the  future  (Another  researcher  can  get  the  same) 

4)  Communicable  -  to  produce  sufficient, 
understandable  documentation  so  the  effort  can  be 
independently  duplicated,  and  so  the  consumer  can 
make  an  informed,  and  perhaps  qualified,  decision  (It 
is  understandable  to  those  who  care) 

Other  objectives  include  the  ability  of  the  process  to  do 
the  following: 

1 )  Describe  the  bounds  of  use  for  the  specified  purpose 

2)  Communicate  the  risk  of  use  for  the  specified 
purpose 

The  necessary  information  when  communicating  the 
results  of  validation  activities  includes,  but  is  not  limited 
to,  data  sources;  referent  sources  and  descriptions;  designs 
of  experiments;  data  and  metadata  for  the  model;  initial 
conditions;  boundary  conditions;  parameters; 
assumptions;  analyses  performed  and  methodologies 
followed;  and  appropriate  uses  of  results. 

The  primary  purpose,  and  importance,  of  conducting 
validation  activities  is  to  assess  the  risk  of  using  an  M&S 
for  a  specific  application  of  use.  The  validation  process 
culminates  in  the  communication  of  that  risk  to  model  and 


simulation  users  and  the  recipients  of  their  data.  This 
includes  determining  that  the  simulation  is  correct  and 
meets  requirements  through  software  engineering  and 
other  processes  but  is  not  limited  to  that.  It  also  includes 
providing  users  with  sufficient  information  to  determine  if 
the  simulation  can  meet  their  needs  as  well  as  determining 
the  simulation’s  capabilities,  limitations,  and  performance 
relative  to  the  real-world  objects  it  simulates. 

3.3  The  validation  of  HBR  models 

The  validation  literature  consists  mainly  of  validation 
approaches,  paradigms,  and  techniques  as  well  as  specific 
validation  applications  and  assessments.  There  is  no 
mechanism  guiding  the  appropriate  selection  of  approach 
and  techniques  in  a  given  M&S  application.  Further,  in 
the  physical  sciences  the  concept  of  valid  models  is 
well-understood;  this  is  not  the  case  in  HBR  modeling.  In 
particular,  these  models  have  inherent  validation 
difficulties  due  to  the  characteristics  of  these  models 
(referents  that  have  poor  computational  underpinnings, 
complexity,  chaotic  effects,  etc.)  and  to  their  desired  uses 
(e.g.,  Course  of  Action  (CO A)  Analysis).  Techniques  for 
validation  will  require  methods  grounded  in  the  larger 
validation,  computational  sciences,  and  experimental 
design  literature  and  apply  them  to  the  growing  field  of 
HBR  model  validation.  Any  technique  applied  in  this 
domain  will  require  an  assessment  of  the  chosen 
conceptual  model,  its  implementation  in  codes,  and  the 
subsequent  simulation  results  once  used. 

3.4  Conceptual  model  validation 

The  conceptual  model  is  the  representation  of  the  content 
and  concept  for  the  model  that  includes  the  logic, 
algorithms,  assumptions,  and  limitations  (Department  of 
Defense  1998).  Verification  ensures  that  the  code 
correctly  captures  this  conceptualization.  In  validation, 
the  conceptual  model  is  compared  against  the  specified 
referent.  In  particular,  the  conceptual  model  must  be  true 
to  within  the  limits  of  acceptability  criteria  in  terms  of  the 
true  statements  within  the  referent.  While  there  may  be 
things  that  are  true  in  the  referent  that  are  not  true  in  the 
conceptual  model,  the  obverse  should  not  occur.  That  is, 
not  true  in  the  conceptual  model  does  not  necessarily 
imply  not  true  in  the  real  system  that  the  referent 
represents.  However,  there  may  be  things  that  are  true  in 
the  real  system  and  in  the  referent  for  that  system  that  are 
not  true  in  the  conceptual  model  because  those  items 
purposely  were  neglected  or  abstracted  out. 

While  initial  assessments  may  find  the  conceptual  model 
to  be  valid,  the  simulation  may  produce  invalid  results 
nevertheless.  This  may  result  from  elements  initially 
deemed  not  important  in  the  model  development, 
incorrect  relationships  between  elements,  inappropriate 
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abstraction  for  the  intended  use,  or  poor  assumptions. 
This  may  especially  be  true  in  systems  where  the 
conceptual  model  reflects  a  referent  based  in  underlying 
theories  of  the  system  without  a  strong  mathematical, 
analytical,  or  logical  description  that  translates  itself  more 
easily  into  code.  This  is  partly  because  programmers  can 
only  code  those  relationships  they  understand  and  in  part 
due  to  the  fact  that  there  are  many  ways  to  describe 
desired  relationships  computationally.  For  instance,  just 
as  there  are  many  possible  rule  sets  for  describing  a  single 
agent  system,  there  are  multiple  ways  to  model  the 
relationship  y  increases  with  x.  Results  validation  may 
uncover  needed  changes  in  the  specification  of  the 
conceptual  model  thereby  uncovering  an  invalid 
conceptual  heretofore  thought  of  as  valid. 

The  testing  of  assumptions  made  in  the  model  may  also 
uncover  previously  undiscovered  defects  in  the  M&S. 
These  assumptions  could  include  seemingly 
inconsequential  assumptions  made  during  coding  efforts 
such  as  the  precision  used  for  n  or  the  simulation  time 
step  or  more  obviously  important  assumptions  like 
whether  the  earth  is  flat  or  spherical  or  the  selected  social 
theory.  Documentation  for  every  assumption  used  in 
developing  and  coding  a  model  is  rarely  complete. 
However,  assumptions’  testing  does  not  require  the 
explicit  identification  of  every  assumption.  Only  those 
assumptions  potentially  affecting  the  use  of  the  M&S 
need  assessment  for  their  impact.  Part  of  the  art  in 
devising  the  validation  analysis  assessing  a  model’s 
assumptions  is  in  recognizing  the  types  of  assumptions 
that  might  be  significant  on  its  use  given  a  description  of 
the  model  and  the  context  of  its  specific  use  and  devising 
tests  to  assess  the  impact  of  the  assumptions  made.  Tests 
might  include  sensitivity  analyses  about  the  assumptions, 
accuracy  assessments  to  ensure  that  the  chosen  precision 
is  sufficient,  or  any  other  appropriate  test.  Thus,  one 
cannot  decouple  the  results  validation  from  validation  of 
the  conceptual  model. 

3.5  Results  validation 

Results  validation  is  only  meaningful  in  the  context  of 
specific  identification  of  what  constitute  valid  results. 
This  is  stressed  both  in  the  W&A  RPG  and  by  Harmon 
and  Youngblood  in  the  importance  of  stating  the 
acceptability  and  validation  criteria  up  front;  i.e.,  the 
necessary  elements  for  using  and  trusting  the  M&S.  That 
is,  stating  up  front  the  necessary  elements  for  using  the 
M&S.  This  is  equivalent  in  the  validation  theory  of 
describing  the  natural  system  or  referent  trajectories 
against  which  M&S  trajectories  will  be  compared  and  the 
validity  relation  that  will  be  used  to  make  the  comparison. 
It  could  include  statistical  comparisons  of  simulation 
output  to  assess  the  real  world  match.  Often  this  is  an 
accuracy  specification  required  to  support  the  intended 


use  of  the  M&S.  Engineering  models  (e.g.,  for  system 
design  and  development  or  for  test  and  evaluation)  require 
predictive  accuracy  most  likely  assessed  using  a  metric 
relation.  On  the  other  hand,  campaign  models  may  only 
require  sufficient  accuracy  to  enable  relative  comparisons 
between  alternative  outcomes  based  on  changes  to  tactics, 
forces,  or  equipment.  Necessary  to  this  assessment  is  the 
determination  of  the  simulation  results  to  be  measured, 
the  material  in  the  referent  against  which  these  results  are 
compared,  the  mechanism  of  comparison,  and  the 
requirements  of  the  results’  acceptability.  Results 
validation  could  run  the  gambit  from  a  state-by-state 
match  to  observed  or  empirical  data  or  with  some 
theoretical  or  posited  expectation  to  an  assessment  that 
the  overall  trends  occurring  in  the  model  match  the 
theory.  In  the  absence  of  this  specification,  the  validator, 
users,  and  subject  matter  experts  will  make  their  own 
implicit  assumptions  of  what  is  required. 

Comparing  simulation  results  to  empirical  or  observed 
data  is  preferable.  While  a  metric  relation  could  be  used 
to  assess  accuracy  (i.e.,  the  delta  between  values),  other 
accuracy  measurements  are  possible  (e.g.,  comparisons  of 
direction,  slope,  or  relative  magnitude).  When  this  kind 
of  data  is  not  explicitly  available,  the  validator  still  needs 
to  assess  whether  the  simulation  output  meets  the  needs  of 
the  intended  use  (e.g.,  can  help  answer  the  analytical 
questions).  In  this  case,  results  validation  relies  on  robust 
test  cases  and  specification  of  expected  results  within  the 
referent  determined  either  from  theory  or  SME  opinion. 

4.  CGM  Overview 

The  CGM  is  a  government-owned,  open  source,  data 
driven  multi-agent  social  simulation.  Actors,  rules,  and 
laws  within  the  model  are  built  upon  social  and 
behavioral  science  theories.  A  modular  framework  is 
used  to  allow  the  incorporation  of  other  social  theories  or 
the  use  of  different  applications  as  the  CGM  grows  in 
maturity.  The  current  implementation  of  the  model  uses 
the  narrative  paradigm,  theory  of  planned  behavior,  and 
Implementation  of  Entity  Cognition  with  Bayesian  Belief 
Networks  (BBN)  to  determine  entity  states. 

4.1  Narrative  paradigm 

The  use  of  the  CGM  requires  understanding  of  the  culture 
in  which  the  scenarios  of  interest  take  place.  Within  the 
model,  cultural  beliefs  of  the  entities  drive  reactions  to 
events  occurring  within  the  scenario  along  with  social 
interactions  between  entities.  To  provide  a  basis  for  the 
connection  between  cultural  factors,  entity  beliefs,  and 
activities,  narrative  theory  plays  a  critical  role  in  the 
development  of  data  in  the  model.  In  narrative  theory, 
people  are  storytellers  and  view  the  world  through  a 
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Figure  3.1:  Theory  of  Planned  Behavior,  By:  leek  Aizen,  2006  (adapted) 
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narrative  lens,  thus  irrational  actions  may  actually  be 
rational  given  their  history  and  culture.  Its  selection  was 
based  on  Fisher’s  argument  (Fisher,  1988)  as  follows: 

1)  people  are  essentially  storytellers; 

2)  reasons  for  decisions  include  history,  culture,  and 
perceptions  about  the  status  and  character  of  the  other 
people  involved  (all  of  which  may  be  subjective  and 
incompletely  understood); 

3)  narrative  rationality  is  based  on  the  probability, 
coherence  and  fidelity  of  the  stories  that  underpin  the 
immediate  decisions  to  be  made;  and 

4)  the  world  is  a  set  of  stories  from  which  each 
individual  chooses  the  ones  that  match  his  or  her 
values  and  beliefs. 

Selection  of  stories  for  use  in  data  development  follow 
Fisher’s  proposal  of  evaluating  stories  based  on  whether 
the  narrative’s  coherence,  probability,  and  fidelity. 
Narrative  coherence  means  the  story  should  make  sense 
structurally,  have  detail  and  characters,  and  should  be  free 
of  surprise.  Narrative  probability  concerns  the  belief  of 
listeners  in  the  truthfulness  of  the  story  irrespective  of  the 
story’s  actual  truthfulness.  Narrative  fidelity  addresses 
the  truthfulness  of  a  story  with  respect  to  cultural  values 
that  include  embedded  values,  relevance  between  the 
story  and  the  values  espoused,  consequences,  consistency, 
and  transcendence. 

4.2  Theory  of  planned  behavior 

The  theory  of  planned  behavior  provides  the  underlying 
basis  for  the  development  of  data  for  entity  intention, 
action,  choice,  and  selection  within  the  CGM.  In  the 


theory  of  planned  behavior  (Figure  3.1)1,  entities  form 
behavioral  intentions  based  on  attitudes,  perception  of 
group  norms,  and  perceived  level  of  control. 

4.3  CGM  Conceptual  Model 

To  Be  Added  in  final  paper  -  Provide  a  description  of  the 
CGM  mathematical  and  logical  implementation  guiding 
the  direction  for  the  validation  effort. 

5.  Challenges 

The  problems  we  face  in  the  current  warfare  environment 
make  the  development  of  HBR  models  sufficient  to 
address  the  problems  of  interest  and  their  validation 
importance.  Having  useful,  credible,  robust  information 
is  critical  for  the  support  of  sound  decision-making. 
However,  limitations  in  the  current  state  of  the  art  create 
challenges.  First,  the  systems  of  interest  are  complex. 
One  of  the  reasons  for  developing  the  models  is  to 
develop  an  understanding  of  the  systems’  behavior  in 
response  to  various  scenarios  that  might  occur.  That  is, 
we  want  to  understand  the  system  of  interest.  However, 
the  social  science  that  forms  the  underpinning  of  these 
models  often  has  multiple,  conflicting  theories  for 
behavior,  complicated  by  variances  in  responses  by 
culture  and  stressor.  This  creates  difficulty  in  model 
development  and  acceptability.  That  is,  our 
understanding  of  the  system  is  limited. 


1  Copyright  Notice:  The  theory  of  planned  behavior  is  in  the 
public  domain.  No  permission  is  needed  to  use  the  theory  in 
research,  to  construct  a  TpB  questionnaire,  or  to  include  an 
original  drawing  of  the  model  in  a  thesis,  dissertation, 
presentation,  poster,  article,  or  book.  However,  if  you  would 
like  to  reproduce  a  published  drawing  of  the  model,  you  need  to 
get  permission  from  the  publisher  who  holds  the  copyright.  You 
may  use  the  drawing  on  this  website  for  non-commercial 
purposes  so  long  as  you  retain  the  copyright  notice.  -  To  Be 
Redrawn 
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Second,  the  systems  of  interest  are  dynamic.  The 
development  and  testing  of  models  requires  data  to 
support  them.  Further,  these  models  also  require  data 
related  to  the  relationships  between  elements  or  entities 
within  the  model.  This  includes  influence  relationships 
between  elements  as  well  as  cause-effect  relationships. 
Not  only  is  obtaining  this  data  difficult,  especially  for  the 
problems  of  interest,  the  data  developed  is  often 
qualitative  vice  quantitative  and  has  an  unknown  valid 
lifetime.  In  particular,  it  is  unknown  whether  the  data 
valid  lifetime  exceeds  the  initial  stressor  events  of 
interest. 

The  third  challenge  is  a  direct  result  of  the  first  two. 
Since  these  M&S  exist  in  a  computer,  necessary  to  the 
model  development  is  a  computational  representation  of 
the  social  theories,  interactions,  and  behaviors  of  interest. 
While  there  are  some  accepted  representations  such  as 
Bayesian  networks,  this  is  far  different  from  the  general 
acceptance  found  in  the  computational  representations 
found  in  the  physical  sciences.  To  create  valid  models, 
both  conceptual  model  and  results  validation  is  required. 
The  validation  of  either  requires  progress  in  both  the 
social  sciences  to  develop  accepted  computational 
representations  as  well  as  measureable  system  responses 
to  events  or  inputs  to  the  system. 

6.  Next  Steps 

The  objective  of  this  project  is  a  repeatable  approach  for 
validating  cultural  behavior  models,  particularly  the 
conceptual  model,  including  risk  measures  and  criteria  for 
assessing  risk  using  the  CGM  as  a  vehicle  for  the 
method’s  implementation.  While  there  are  many 
challenges  in  HBR  modeling,  making  progress  in 
techniques  for  the  M&S  of  HBR  and  in  developing 
methods  validating  those  M&S  is  necessary.  The  next 
steps  in  this  project  are  to  continue  evaluation  of  the 
CGM  conceptual  model.  Critical  to  the  effective  use  of 
M&S  is  the  understanding  of  the  risk  in  that  use  for  a 
specific  problem  of  interest.  This  is  the  key  goal  for 
validation.  The  understanding  of  the  risk  in  using  a 
simulation  for  a  specified  use  is  a  core  area  of  research  for 
this  work. 

There  are  two  components  of  risk  in  general  (Defense 
Acquisition  University  2003): 

1 .  The  probability  or  likelihood  of  achieving  (not 
achieving)  a  given  outcome 

2.  The  consequences  of  achieving  (not  achieving)  a 
given  outcome 

There  is  higher  risk  with  a  higher  likelihood  or  with 
significant  consequences.  Risk  assessment  includes  both 
the  identification  of  risk  (determination  of  outcomes)  and 


the  analysis  of  risk  (determination  of  probability  and 
consequence  of  an  outcome).  It  is  in  this  latter  aspect  that 
M&S  often  plays  a  role.  That  is,  the  intended  use  for  an 
M&S  is  to  identify  and  help  to  mitigate  risk,  identified  as 
part  of  some  specified  objective.  However,  the  use  of 
M&S  in  this  analysis  poses  an  inherent  source  of  risk. 
The  sources  of  risk  could  lie  in  the  development  of  the 
model,  development  risk,  or  in  the  running  of  the 
simulation,  operational  risk  (Modeling  and  Simulation 
Coordination  Office  2004b).  Development  risk  is  that  the 
model  does  not  meet  the  requirements  for  its  intended  use. 
Operational  risk  is  that  the  M&S  exhibits  insufficient 
accuracy  to  provided  needed  information.  The  V&V 
process  addresses  both  these  risk  areas.  When 
considering  intended  use,  risk  can  be  described  generally 
using  the  three  familiar  error  types: 

1.  Type  I  Error:  Reject  correct  information;  the 

information  provided  by  the  M&S  is  not  used  in 
solving  the  problem  even  though  the  information 
provided  is  correct. 

2.  Type  II  Error:  Accept  incorrect  information;  the 

information  provided  by  the  M&S  is  used  in  solving 
the  problem,  however,  the  information  provided  is 
incorrect. 

3.  Type  III  Error:  Solve  the  wrong  problem;  the 

information  provided  by  the  M&S  is  irrelevant  to  the 
actual  problem  to  be  solved. 

Validation  primarily  assesses  the  Type  II  error.  When 
assessing  the  consequences  of  using  incorrect  data  in  a 
decision,  considerations  include  who  is  affected,  the 
severity  of  the  effect,  and  the  visibility  of  the 
consequences.  Development  risk  assesses  the  effect  of 
not  meeting  requirements,  the  likelihood  of  a  deficiency, 
and  the  probability  that  a  deficiency  will  cause  the  M&S 
not  to  meet  requirements.  These  assessments  drive 
toward  the  fundamental  assessment  of  whether  the  M&S 
support  the  intended  use.  Operational  risk  assesses  the 
probability  of  making  an  incorrect  decision,  the  effect  and 
visibility  of  making  an  incorrect  decision,  and  specific 
user  considerations. 
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